Stability Verification Method Of Stable Machine Us High Defense Server In Long Connection Business

american high-defense server, long connection, stability verification, websocket, tcp long connection, stress test, network tuning">

1.

goals and preparation overview

goal: verify the stability and availability of the us high-defense server under long connection (tcp/websocket/http2 long polling, etc.) services.
preparation items: list the test ip, port, domain name, certificate (if any), business protocol, concurrent connection target, duration target (for example, 72 hours of stability), and performance monitoring access (prometheus/grafana).

2.

environment setup: server configuration and dependencies

step 1: deploy the business service on the target anti-ddos pro c segment or independent ip, and confirm the service listening port and protocol (such as ws:// or wss://).
step 2: install necessary tools: ss (iproute2), netstat, tcpdump, iftop, htop, sysstat, prometheus node_exporter; install pressure tools (wrk2, tsung, h2load, gattling or custom go client).

3.

basic network and system configuration checks

command practice: sysctl -a | grep net.ipv4.tcp_tw_reuse, check and adjust the core parameters: net.ipv4.tcp_tw_reuse=1, net.ipv4.tcp_tw_recycle=0 (if applicable), net.ipv4.tcp_fin_timeout=30.
adjust the file handle: ulimit -n 200000 and persist it in /etc/security/limits.conf. check epoll/thread pool settings and maximum number of processes (/proc/sys/fs/file-max).

4.

long connection application layer settings

websocket/http2 applications need to enable the heartbeat/ping mechanism. it is recommended to configure the heartbeat interval as an example: the server initiates a heartbeat every 30 seconds, and the client timeout reconnection threshold is set to 3 unresponsive times.
set the connection timeout and maximum idle time to avoid the default timeout of the load balancer (such as nginx/haproxy) and cut off the connection. adjust nginx proxy_read_timeout and proxy_send_timeout to at least 120s or higher.

5.

test case design: scenarios and indicators

define scenarios: concurrent long connection establishment (peak), continuous connection stability (whether it is dropped after being idle for a long time), sudden concurrency growth (staircase load), performance when packet loss/latency worsens (network jitter).
key indicators: connection success rate, connection disconnection rate, average single connection delay, p95/p99 delay, number of reconnections, cpu/memory/network traffic, number of sockets (ss -s).

6.

pressure tool and script practice (example)

use wrk2 or a custom go client to simulate long connections: example go: use gorilla/websocket to establish n persistent connections, send heartbeats cyclically and record disconnection events.
the concurrent script must control the number of connections, heartbeat frequency, and sending traffic size, and record new connections, disconnections, and reconnections every second to a file for subsequent analysis.

7.

network fault injection and jitter testing steps

use the tc command to inject delay and packet loss: tc qdisc add dev eth0 root netem delay 100ms loss 1%; observe the reconnection and timeout performance of the server and client under different packet loss/delay.
gradually increase the packet loss rate and delay, record the disconnection rate curve, and determine whether the high-defense strategy affects the stability of long connections (such as active disconnection, connection restrictions, and timeout policies).

8.

high-defense feature verification: connection restrictions and cleaning behavior

communicate with the high-defense service provider to confirm the cleaning threshold (such as syn/connection rate threshold, concurrent connection threshold), gradually approach the threshold during the test, and observe whether active disconnection or traffic cleaning occurs.
if a whitelist ip or port can be configured, test the difference before and after the whitelist is enabled to confirm the impact of the whitelist on long connections.

9.

monitoring and log collection configuration details

deploy node_exporter + cadvisor (if containerized) to collect host/process indicators; record connection open/close/heartbeats/error logs at the application layer and send them to elk or loki.
set up the grafana panel: socket_count, new_connections/s, disconnects/s, cpu, net_rx/tx, tcp_retransmits. configure prometheus alarm rules: the disconnection rate >0.5%/5min triggers an alarm.

10.

fault recurrence and step-by-step troubleshooting process

if stability problems are found, troubleshoot according to priority: 1) monitor whether resources are exhausted (file descriptors, cpu); 2) check whether the firewall/high defense policy is triggered; 3) use tcpdump to capture packets to compare the client/server handshake and heartbeat; 4) check application logs and gc/exception stacks.
example commands: ss -tanp | grep :port; tcpdump -i eth0 host client_ip and port port -w capture.pcap.

11.

long-term stability verification process (example 72 hours)

step 1: establish a baseline (24-hour low load monitoring) and confirm that there are no abnormalities. step 2: enter the stress period (48 hours), maintain long connections and heartbeats concurrently as expected, and record all indicators.
step 3: inject jitter (network delay/packet loss) and sudden short-term concurrency growth (10-30%) in the intermediate stage, and record service degradation or disconnection. finally, collect all logs, capture packets, and monitor charts to generate reports.

12.

regression and optimization suggestion list

common optimizations: increase file handles, adjust tcp_keepalive time, disable tcp_tw_recycle, optimize application heartbeat and reconnection strategies, extend timeouts at the proxy layer, and reasonably configure high-defense thresholds and whitelists.
record the effects of each optimization (changes in disconnection rate, cpu/memory changes) and include them in continuous integration or operation and maintenance runbooks.

13.

q: how to do stress testing without affecting real users?

answer: use mirror traffic or use request playback sampled from production in the test environment; if you need to test in production, first use a small portion of whitelist ips or grayscale traffic, limit the test ip access ratio, and set a whitelist or higher threshold at the high-defense location. in addition, use non-business critical periods and detailed alerts, and ensure rollback plans and methods to quickly block test traffic (such as temporarily modifying firewall rules).

14.

q: what should i do if a large number of time_wait and connection exhaustion occur?

q: what should i do if a large number of time_wait and connection exhaustion occur? answer: first confirm the source of time_wait through netstat/ss. adjust the strategy: enable connection reuse (keep-alive) on the client or increase the port range of short connections; set net.ipv4.tcp_tw_reuse=1 on the server and reasonably reduce tcp_fin_timeout, increase the upper limit of file handles (ulimit -n), and optimize the application layer reuse logic to reduce frequent connection establishment.

15.

q: the high-defense strategy may misjudge long connections as attacks. how to avoid this?

answer: cooperate with high-defense vendors to explain the business characteristics (large number of persistent connections, heartbeat frequency), strive to add business ips or ports to whitelists or special rules, and adjust cleaning thresholds (syn/connection rate, etc.). at the same time, a small randomization of heartbeat desynchronization is added to the client to avoid triggering thresholds for centralized synchronization behavior. record and submit the pcap and monitoring curve of the trigger event to facilitate debugging by the other party.